1. Field of the Invention
This invention is related to the general field of academic assessment testing, and within that field to the task of selecting test taker responses appropriate to be exemplar responses for the score points in open-ended (non-objective) assessment items, commonly referred to as xe2x80x9crange findingxe2x80x9d. Although the methods described herein may be useful in other related tasks, the most common application is likely to be range finding to provide exemplar responses for training the readers or scorers of the responses to the open-ended test items.
2. Description of Related Art
While multiple-choice and other objective assessment items have proven to be cost effective and generally accurate for many tests, open-ended or non-objective items are often specified for a variety of tests. Typically such open-ended items must be scored by human xe2x80x9creadersxe2x80x9d; that is, each response is read by one or more trained readers who assign appropriate scores to the response.
It is necessary to train multiple readers to give equivalent scores to assessment responses. Typically training is done by showing readers the rules under which the scores are given and sample responses representing all of the possible xe2x80x9cscore pointsxe2x80x9d. The rules are embodied in a xe2x80x9crubricxe2x80x9d, while the sample responses are included in xe2x80x9canchor setsxe2x80x9d, xe2x80x9ctraining setsxe2x80x9d, and/or xe2x80x9cqualifying setsxe2x80x9d of items. So-called xe2x80x9canchor setsxe2x80x9d typically consist of sample responses selected to represent an xe2x80x9cexemplarxe2x80x9d response for each score point. xe2x80x9cTraining setsxe2x80x9d include exemplar responses selected to teach readers how to properly assign all possible scores. xe2x80x9cQualifying setsxe2x80x9d consist of responses selected to verify that any reader can appropriately assign the proper score point to a student response after having been trained.
xe2x80x9cRange findingxe2x80x9d is a term that refers to selecting student responses appropriate to be the sample responses for training and related materials including, but not limited to, anchor sets, training sets, and qualifying sets. The term xe2x80x9crange findingxe2x80x9d reflects the fact that exemplar responses must be located or xe2x80x9cfoundxe2x80x9d representing the full range of scores. Range finding is typically performed on a sampling of unscored student responses. Often the sample of unscored materials is a subset of student responses xe2x80x9cpulledxe2x80x9d from a live administration of a test. In some cases the sample will be from a smaller xe2x80x9cfield testxe2x80x9d administration designed to check the test before it is put into operational status.
When items are scored as credit/no credit (or other dichotomous scoring), exemplar samples can generally be found quickly. Many open-ended items, however, are scored on scales, typically with 4 to 6 score points. This significantly complicates the range finding since exemplar samples must be found for all points. In addition, many tests are designed to identify students that meet high standards, a result of which may be that high-end exemplars may be rare or even non-existent within the particular sample of student responses being examined.
Range finding has been traditionally performed by a team of content experts and scoring specialists working together with the paper responses. Members of the team go through unscored test materials looking for sample responses representing each score point. When one member xe2x80x9cfindsxe2x80x9d what he or she believes to be an appropriate student response for a particular score point, at least one other member of the team confirms the score point for that response. This process can be tedious. Often the sample of papers consists of student responses clustered around one or two score points so that team members spend the bulk of their time looking at samples of the same score points again and again. As a result, they often resort to quickly glancing at responses xe2x80x9clookingxe2x80x9d for papers that might be examples of the higher score points. Sometimes they need to go through each and every student response in the sample, and even then may not find enough samples of each score point for training and qualifying readers.
The problem of scarce high score points is well known, and methods have been found methods to alleviate the problem. When large test administrations are the source of the sample of students, administrators will pick some student responses from traditionally high scoring schools or classrooms of honors students to include in the sample. When the sample is based on a field test, administrators will try to recruit participation by at least one or more high performing schools.
In assessment testing, answer documents may contain both pages that are processed by Optical Mark Reading (OMR), and separate pages that contain responses (or answers) to open-ended assessments, such as hand-written essays or other responses that can have a range of score points. When completed test documents are received for the scoring process, they are prepared for optical scanning and processed by an Optical Mark Scanner. In the scanning progress, the pages with OMR responses are processed and separated from the pages with open-ended responses. The pages with open-ended responses are divided into xe2x80x9cpacketsxe2x80x9d of materials to be read and scored. These packets are passed on to trained readers so that the open-ended responses can be scored.
When range finding is required to locate exemplar responses for the range of score points, some documents are diverted from the normal processing and become the sample of documents that are reviewed for range finding. When entire test documents are used for range finding, some documents are diverted to the range finding process prior to scanning and are examined by the range finding team. The responses selected as score point exemplars are then typically photocopied, and the copies are used for training materials. Once they have been reviewed for the range finding process, the diverted documents are put back into the production process for scanning.
The range finding is also sometimes done by diverting just a sample of pages with open-ended responses, rather than the entire test document. After scanning the test document, some pages containing open-ended responses are diverted to the range finding team for review as exemplars of score points. Again, the responses selected as exemplars are typically copied and the copies are used for training. Once they have been through the range finding process, the diverted sheets are put into packets and the packets are forwarded to readers for scoring.
When the entire test documents are large, the process of diverting only the open-ended response pages has the advantage of providing the range finding team with less paper. However, because the sheets containing student identification are typically separated during the scanning process, it would be difficult to locate student responses from students in specific schools or to locate responses from specific students, if it were desired to use that information to target sets of responses to examine for particular score points.
The current invention consists of methods to expedite and simplify the range finding process. First, the invention utilizes digitized representations, such as scanned images, of test taker answers to review for exemplars of score points, rather than review of the paper documents containing the student answers. Secondly, the invention uses a method of identifying targeted subsets of individual student answers which are likely to be exemplary of any given score point. In the preferred implementation of this invention, digital representations of the answers are linked to the student identity and some student performance indicator so that a computer system can determine a targeted subset. Alternatively, the digital representations could be linked to a performance indicator without utilization of the student identity. The invention also allows for expanding the sample of answers available to range finding, storing evaluation data, and sequencing answers.
Use of Digitized Representation of Open-Ended Responses
Range finding is significantly enhanced by the use of digitized representations of student responses. The most common of such digital representation is an optically scanned image of a student""s written response, although the digitized images could be created by keyed input, OCR, voice recognition or similar interface. In this environment, the reader-team members examine the digitized representation of the open-ended responses on computer screens rather than going through hundreds or thousands of student responses on paper. Scores assigned by readers are immediately recorded and saved within the computer system. The benefits of using digitized representation of open-ended responses for range finding parallel the significant benefits of using digitized representations of open-ended responses for scoring, as shown in U.S. Pat. No. 5,672,060 (Poor).
Use of Targeted Subset of Sample
Range finding generally starts with a thorough review of the scoring rubric and then an examination of a random or arbitrarily selected subset of responses from the sample. This examination may be done individually or as a group and generally leads to identification of samples of some score points. As the process continues, more student responses are given scores. At some time, there are sufficient sample responses for some score points (generally the middle or lower points within the scale), but not for others (generally the higher score points, but sometimes also the lowest). From this situation, range finding can be made much more efficient by providing the team subsets of the sample which are more likely to contain the score points which are under represented.
It is known that different academic assessments and other indicators of student performance are often highly correlated. The task of finding examples of specific score points, then, can be reduced by providing the team members with subsets of the sample which represent individual students who have shown, for example, high scores on a related assessment or other academic performance data.
Using traditional paper techniques, this can be done by providing the expert reader team with reports ranking student performance for all students in the sample. Members could then find any given student""s response by looking for that student""s document within the appropriate test administration group of documents (often the student""s school and classroom). In seeking responses to review for exemplars of the highest score point, for example, the reader team could select responses from student""s with a high academic performance in the ranking.
Using on-screen presentation of the student responses, however, there are at least four methods whereby expert readers can examine targeted subsets of responses from the sample:
readers can be presented with a subset of sample responses selected on a range of performance of students in the sample on an appropriate academic performance indicator;
readers can be presented with student responses that are ordered or ranked by the student scores on a selected academic indicator;
readers can be presented with a subset of sample responses presented for evaluation ordered or ranked by the student scores on an academic indicator which includes only those students whose scores fell within a specified range of scores;
readers can be presented with a specific student""s response where that student has previously been identified based on performance on a related item or items.
Any of these methods can greatly expedite range finding, and this list is not intended to be exhaustive of such methods. Each of these methods of targeted examination of student responses significantly improves the likelihood of finding exemplars of needed score points and thereby significantly reduces the time, effort, and cost of range finding.
Academic Performance Indicators:
Although other indicators of student performance can be used, the following indicators of academic performance are often available at little or no cost and can be used individually or in combination:
Score on objective test items, such as multiple choice items: With an increasing number of assessments including both multiple-choice and open-ended assessment items, the scores on the multiple-choice items can be used to rate student performance on each section of the test. These scores can then appropriately be used to target selection of open-ended items within each section.
Score on other assessment: Using standard computer techniques, student scores from earlier assessments can be matched to each student""s current assessment. These scores can then appropriately be used to target selection of related open-ended items within the assessment.
Course performance: Student grades in current or prior courses can be used. Such grades can be matched to the student using standard computer techniques, or the grades can be entered directly onto the student answer document. These grades can then be used to target selection of related open-ended items within the assessment.
Teacher performance rating: Teachers can be asked to indicate the relative performance rating of each student on each section within the assessment. Such ratings can be inexpensively collected directly on the student response document. These ratings can then be used to target selection of related open-ended items within the assessment.
Student performance rating: Students can be asked to rate their own performance, on specific open-ended assessment items or on sections within the assessment. Such ratings can be inexpensively collected directly on the student response document. These ratings can then be used to target selection of related open-ended items within the assessment.
This list is not intended to be exhaustive of the indicators which may be used to predict whether the response is more likely to be graded within a limited range of score points for the assessment than the response of other test takers.
Expanding the Sample Available for Range Finding
In most assessments, students complete their responses on specially designed answer documents suitable for processing by Optical Mark Reading (OMR). In traditional paper techniques, the range finding is performed before student answer documents are scanned or after the documents are scanned and before they are sent for scoring. Often, the sample for range finding is diverted to the team doing range finding while the balance of the answer documents go through normal production processing. As a result, it is extremely difficult to get additional student responses to the team if the original sample is not sufficient.
By using the digitized representation of student responses, all documents go through the same process: a single scanning process is used to capture both the OMR data (identifying information and multiple-choice items) and the images of the student responses to open-ended responses. The original sample for the range finding team is processed first so that range finding can begin as soon as possible. While the range finding team is conducting range finding, however, additional documents are being scanned and therefore become available for range finding. If necessary to find enough of the needed exemplars, the range finding team can complete its task having responses from most or all of the students tested.